Search Results: "rudi"

8 May 2017

Lars Wirzenius: Ick2 design discussion

Recently, Daniel visited us in Helsinki. In addition to enjoying local food and scenerey, we spent some time together in front of a whiteboard to sketch out designs for Ick2. Ick is my continuous integration system, and it's all Daniel's fault for suggesting the name. Ahem. I am currently using the first generation of Ick and it is a rigid, cumbersome, and fragile thing. It works well enough that I don't miss Jenkins, but I would like something better. That's the second generation of Ick, or Ick2, and that's what we discussed with Daniel. Where pretty much everything in Ick1 is hardcoded, everything in Ick2 will be user-configurable. It's my last, best chance to go completely overboard in the second system syndrome manner. Where Ick1 was written in a feverish two-week hacking session, rushed because my Jenkins install at the time had broken one time too many, we're taking our time with Ick2. Slow and careful is the tune this time around. Our "minimum viable product" or MVP for Ick2 is defined like this:
Ick2 builds static websites from source in a git repository, using ikiwiki, and published to a web server using rsync. A change to the git repository triggers a new build. It can handle many separate websites, and if given enough worker machines, can build many of them concurrently.
This is a real task, and something we already do with Ick1 at work. It's a reasonable first step for the new program. Some decisions we made: The initial pipelines will be basically like this, but expressed in some way by the user:
  1. Clone the source repoistory.
  2. Run ikiwiki --build to build the website.
  3. Run rsync to publish the website on a server.
Assumptions: Actions the Ick2 controller API needs to support: A sketch API: Sequence: The next step will probably be to sketch a yarn test suite of the API and implement a rudimentary one.

7 November 2016

Reproducible builds folks: Reproducible Builds: week 80 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday October 30 and Saturday November 5 2016: Upcoming events Reproducible work in other projects Bugs filed Reviews of unreproducible packages 81 package reviews have been added, 14 have been updated and 43 have been removed in this week, adding to our knowledge about identified issues. 3 issue types have been updated: 1 issue type has been removed: 1 issue type has been updated: Weekly QA work During of reproducibility testing, some FTBFS bugs have been detected and reported by: diffoscope development buildinfo.debian.net development tests.reproducible-builds.org Reproducible Debian: Misc. Also with thanks to Profitbricks sponsoring the "hardware" resources, Holger created a 13 core machine with 24GB RAM and 100GB SSD based storage so that Ximin can do further tests and development on GCC and other software on a fast machine. This week's edition was written by Chris Lamb, Ximin Luo, Vagrant Cascadian, Holger Levsen and reviewed by a bunch of Reproducible Builds folks on IRC.

24 October 2016

Reproducible builds folks: Reproducible Builds: week 78 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday October 16 and Saturday October 22 2016: Media coverage Upcoming events buildinfo.debian.net In order to build packages reproducibly, you not only need identical sources but also some external definition of the environment used for a particular build. This definition includes the inputs and the outputs and, in the Debian case, are available in a $package_$architecture_$version.buildinfo file. We anticipate the next dpkg upload to sid will create .buildinfo files by default. Whilst it's clear that we also need to teach dak to deal with them (#763822) its not actually clear how to handle .buildinfo files after dak has processed them and how to make them available to the world. To this end, Chris Lamb has started development on a proof-of-concept .buildinfo server to see what issues arise. Source Reproducible work in other projects Packages reviewed and fixed, and bugs filed Reviews of unreproducible packages 99 package reviews have been added, 3 have been updated and 6 have been removed in this week, adding to our knowledge about identified issues. 6 issue types have been added: Weekly QA work During of reproducibility testing, some FTBFS bugs have been detected and reported by: diffoscope development tests.reproducible-builds.org Misc. Our poll to find a good time for an IRC meeting is still running until Tuesday, October 25st; please reply as soon as possible. We need a logo! Some ideas and requirements for a Reproducible Builds logo have been documented in the wiki. Contributions very welcome, even if simply by forwarding this information. This week's edition was written by Chris Lamb & Holger Levsen and reviewed by a bunch of Reproducible Builds folks on IRC.

5 May 2016

Sean Whitton: dh_make_elpa & dh_elpa_test

I recently completed and released some work on Debian s tooling for packaging Emacs Lisp addons to GNU Emacs. Emacs grew a package manager called package.el a few years ago, and last year David Bremner wrote the dh_elpa tool to simplify packaging addons for Debian by leveraging package.el features. Packaging a series of addons for Debian left me with a wishlist of features for dh_elpa and I was recently able to implement them. Debian tooling generally uses Perl, a language I didn t know before starting on this project. I was fortunate enough to receive a free review copy of Perl 5 by Example when I attended a meeting of the Bay Area Linux Users Group while I was visiting San Francisco a few months ago. I accepted the book with the intent of doing this work. dh_make_elpa dh_make_elpa (at present available from Debian experimental) is a Perl script to convert a git repository cloned from the upstream of an Emacs Lisp addon to a rudimentary Debian package. It performs a lot of guesswork, and its simple heuristics are something I hope to improve on. Since I am new to object-oriented program design in Perl and I wanted to leverage object-oriented Debian tooling library code, I took the structure of my project from dh_make_perl. In this manner I found it easy and pleasant to write a maintainable script. dh_elpa_test A lot of Emacs Lisp addon packages use a program called Cask to manage the Emacs Lisp dependencies needed to run their test suites. That meant that dh_auto_test often fails to run Emacs Lisp addon package test suites. Since the Debian packaging toolchain already has advanced dependency management, it s undesirable to involve Cask in the package build pipeline if it can be avoided. I had been copying and pasting the code needed to make the tests run in our environment to the debian/rules files of each package whose test suite I wanted to run. dh_elpa_test tries to detect Emacs Lisp addon package test suites and run them with the workarounds needed in our environment. This avoids boilerplate in debian/rules. dh_elpa_test also disables dh_auto_test to avoid a inadvertent Cask invocation. Future & acknowledgements My hope for this work was to make it easier and faster to package Emacs Lisp addon packages for Debian, for my own sake and for anyone new who is interested in joining the pkg-emacsen team. In the future, I want to have dh_elpa_test generate an autopkgtest definition so that a Testsuite: pkg-emacsen line in debian/control is enough to have an Emacs Lisp addon package test suite run on Debian CI. I m very grateful to David Bremner for reviewing and supporting this work, and also for supporting my Emacs Lisp addon packaging work more generally.

28 March 2016

Lucy Wayland: Stuffed Butternut Squash

This is a fusion recipe from a rather bland just stuff it with ricotta recipe I saw, David Scott s The Peniless Vegetarian , and my own mutations on those themes. I can t give you exact quantities, just make a little more than you will make the hollowed mound (grin), and the rest will make an excellent pasta sauce. Ingredients For an average sized butternut squash, you will need:
1 onion (I prefer red)
3 cloves of garlic
1 capsicum pepper (I prefer green, my ex- preferred red)
Some red lentils
Optional green or brown lentils for texture and flavour. I used some puy
The lentil quantity is hard to estimate, but I ratio 4 red to 1 optional.
Roughly one handful of chopped mushrooms i.e. when chopped, it is one handful
1 tin tinned tomatoes
Some tomato puree
A generous amout of garam masala garam masala is what brings out the flavout in lentils
Some paprike
Optional chilli if using chilli, I recommend fresh of course.
Optional Balsamic vinegar
Optional Marmite Preperation of the Squash
1. Cut the butternut squash in half, length ways. This is very hard, you will need a good large knife, and may require you jumping up and down into the air. This is the second most hard of the procedure. 2. For each half, scoop out the seeds, and pare back the bowl till it is no longer overly fibrous. Discard this, or find a use for the seeds. 3. For each half, scoop a channel of the softer flesh up from the baisin up near the top. This has to be done by feel, is hard and thankless work. Also experimentation required. Reserve this flesh. Preperation of the Filling This is just basically a nice lentil sauce that can be used with pasta, rice, toast etc. Important: this is not a stir fry, but a largish, heavy bottom pan is recommended. 1. Finely peel then chopp the onions and the garlic. Chopp the chillis if used (I am a chilli gal). Please observe Chilli Protocol[0] 2. Wash and chop the pepper and mushrooms. Not finely diced, but not crudite-sized slices. Remember that peppers shrivel down a little, mushrooms a lot. 3. Start frying the onions for a while in some oil (I prefer olive, but others are acceptible), until they just about to go translucent. Then add the garlic and optional chillis until the garlic is just cooking nicely. 4. Add the spices, turn over until all the containts of the pan are covered, and cook for another 30 seconds or so. Then add the tinned tomato, and then add half a can of cold water water which rinsed the tin out with. Stir this around, and make sure it is now at just at a simmer or pre-simmer. 5. Add the lentils. You want 0.5-1 cm of water above the lentils when you have added and stirred. Let these cook and expand for about 5 mins, stirring all the while, all the lentils will stick to the bottom. 6. Add the pepper, mushroom, reserved squash flesh, and optional dash of balsamic vinegar, and half a tea spoon of marmite. Cook and stir until the pepper goes soft. This is the hard part. Add boiling water if really too thick, or some tomato puree if too thin. There is no hard science to this, you want at the end of 10 minutes or so something resembling the thickness in texture of a stiff bolognaise sauce. Assembly
1. Have a baking tray. Whether you prefer to grease, line with foil, or line with baking parchment is up to you. I prefer baking parchment. 2. Stuff those two halves of butternut squash with that sauce you made. It should make a mound of about 1cm about the level. If you feel extravagent, and are not vegan, sprinkle a little grated cheese on top. 3. Place in a pre-heated oven of 200oC. Cooking time should be about 20 mins, but larger ones take longer. The acid test is to briefly take them out, and prod the lower side with a fork. It should go through the skin with little resistance. When ready, serve. It s really a dish in itself, but some people might like a bit of salad, or maybe a light green risotto.

26 March 2016

Dirk Eddelbuettel: Rcpp 0.12.4: And another one

The fourth update in the 0.12.* series of Rcpp has now arrived on the CRAN network for GNU R, and has just been pushed to Debian as well. This follows four days of idleness in the incoming/ directory: Word is that the tireless CRAN maintainers were traveling and only covering simpler packages. The 0.12.4 release follows the 0.12.0 release from late July, the 0.12.1 release in September, the 0.12.2 release in November, and the 0.12.3 release in January -- making it the eight release at the steady bi-montly release frequency. As before, this release is more of a maintenance release addressing a number of small bugs, nuisances or documentation issues without adding any major new features. Rcpp has become the most popular way of enhancing GNU R with C or C++ code. As of today, 615 packages on CRAN depend on Rcpp for making analytical code go faster and further. That is up by more than sixty packages from the last release in January! As during the last few releases, we have new first-time contributors. Kirill Mueller extended Nullable<> to const objects. James "coatless" Balamuta helped with a much-needed update to the Rcpp FAQ concerning recommendations for OS X installations. Colin Gillespie corrected another (small) vignette issue. Contributions from the rest of the gang as well as by-now regular contributors such as Nathan or Dan are detailed below.
Changes in Rcpp version 0.12.4 (2016-03-22)
  • Changes in Rcpp API:
    • New accessors as() and clone() were added to the Nullable class (Dan in PR #423 closing #421)
    • The Nullable<>::operator SEXP() and Nullable<>::get() now also work for const objects (Kirill Mueller in PR #417).
    • A subsetting error was fixed (Qiang via #432 closing #431).
  • Changes in Rcpp Sugar:
    • Added new Sugar function median() (Nathan in PR #425 closing #424)
    • Added new Sugar function cbind() (Nathan in PR #447 closing #407)
  • Changes in Rcpp Attributes:
    • A plugin for C++14 was added (Dan in PR #427)
  • Changes in Rcpp Documentation:
    • An entry was added to the Rcpp-FAQ vignette describing the required packages for vignette building (#422).
    • Use on OS X was further detailed (James Balamuta in #433 with further review by Bob Rudis).
    • An entry was added concerning the hard-code limit of arguments to some constructor and function (cf #435).
    • The Rcpp-FAQ vignette now contains a table of content.
    • Typos and indentation were corrected in the Rcpp Sugar vignette (#445 by Colin Gillespie).
Thanks to CRANberries, you can also look at a diff to the previous release. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads page, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

8 January 2016

Dirk Eddelbuettel: AsioHeaders 1.11.0-1

Making it easier to use C++ in R has been a really nice and rewarding side effect of the work on Rcpp. One really good idea came from something Jay Emerson, Michael Kane and had I kicked about for way too long: shipping Boost headers for easier use by R and CRAN packages (such as their family of packages around bigmemory). The key idea here is headers: Experienced C++ authors such as library writers can organise C++ code in such a way that one can (almost always) get by without any linking. Which makes deployment so much easier in most use cases, and surely also with R which knows how to set an include path. So after years of "yes we really should", Jay deserves the credit for actually sitting down almost three years ago and creating the first version. So at the end of January 2013, we released BH version 1.51.0-0. By now the package is used fifty-four other CRAN packages --- clearly a much stronger uptake then we ever expected. I took over maintenance at some point, and we are currently in-line with the most recent Boost release 1.60.0. But some people were always lusting after (some) parts of Boost which also require linking. For some libraries (such as Boost.Date_Time in my RcppBDT package) you can set a #define to not require linking (and forego some string formatting or parsing we get from R anyway). Some others are trickier; Boost Regex is one example. I do not think you can use it without linking (but then C++11 helps and finally brings regular expression in the default library). Another library which the systems / network programmers at work would frequently rely upon is Boost.Asio, a cross-platform C++ library for network and low-level I/O programming. It was already used on CRAN by the iptools package by Bob Rudis and Oliver Keyes which shied away from building on Windoze (while 'doze and Boost do get along, but the CRAN system makes it a little more involved). A couple of days ago, I somehow learned that the Asio library actually comes in two flavours: the well-known Boost.Asio (requiring linking), and also as a header-only standalone library! Well, well, well -- so I ran that by Bob and Oliver asking if that would be useful. And the response was a resounding and pretty much immediate Hell, Yes!. So I wrapped it in a package, told Bob about it, who tested it and gave the thumbs up. And after a slightly longer-than-usual on-boarding, the package is now on CRAN. As network (and systems) programming is not entirely uncommon even at CRAN, I hope that this package may find a few more use cases. A new version of iptools should be forthcoming "shortly", including for the first time a Windows build thanks to this package. The indefatigable Martin Morgan already spotted our GitHub repo and scored the first fork. Comments and suggestions about AsioHeaders are welcome via the issue tracker at the GitHub GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

4 December 2015

Joachim Breitner: Reproducible Builds World Summit

This week, I was attending the first Reproducible Builds World Summit in Athens. A while ago, I have fixed some reproducibility bugs in the Haskell compiler GHC (parts of #4012), and that got me a ticket to this fully sponsored and nicely organised event. This was the first event I attended that had a dedicated and professional moderator, Gunner from Aspiration, who came up with a suitable on-the-fly-schedule, ensured the time was structured by shorter plenary rounds with all ~40 attendees as well as longer working sessions in groups of usually at most 10 people, motivated people to run ( facilitate ) these working groups as well as ensured that any results were noted and collected. This definitely makes a difference to the otherwise unstructured let s all sit in a big circle and endlessly discuss things or let s all just hack on what we want to hack on right now structure of less professionally organised meetings. But after 1 days of talking, a certain desire to hack and get work done could be felt throughout the group, so most of us were happy to be allowed to make use of the laptop on Wednesday afternoon. At least as far as possible the wireless was not expecting this crowd, it seems. Besides the few patches above, my involvement in the project is rather rudimentary, so I tried to contribute as far as possible during the event itself. This was not always easy, as many of the working sessions were either self-reflecting or forward-facing: What problems do we see? What should we get done next? How do we talk to users/upstream/press/doners? Where possible, I tried to at least be helpful in the discussion. During the hacking sessions, I found it most useful to contribute to diffoscope, a program to recursively and deeply show the difference between files of any format, so there are now patches to implement a multi-file HTML output, to more efficiently include large diffs, and more importantly I assisted Lunar in making diffoscope multi-threaded. In Python this is not a very great experience; I guess I am spoiled by Haskell (as it is often the case). I enjoyed the group; a bit like DebConf but with many new faced from similarly-minded, but different projects. It was, however, again a Free Software event with an embarrassing low number of female participants. I liked discussing a proof of G del s first incompleteness theorem during a skill exchange session. Unfortunately, I had to leave very timely after the event on Thursday evening, in order to attend the Haskell in Leipzig workshop.

11 November 2015

Mike Gabriel: My FLOSS activities in October 2015

October 2015 has been mainly dedicated to contracted/payed work. Only a few issues I could address during the last month: Arctica Project While having a week off from work, I managed to get Arctica Greeter to build on non-Ubuntu systems. The issue was very simple. The build crashed during the test suite run and it was caused by the XDG_DATA_DIRS variable not being set in my clean build environment. Furthermore, I added various more session type icons to Arctica Greeter (XFCE, LXDE, MATE, OpenBox, TWM, Default X11 Session, etc.) and also rebased the Arctica Greeter code base against all recent commits found in Unity Greeter for Ubuntu 15.10 / upcoming 16.04. Together with Ulrich Sibiller, I continued our work on the new Xinerama implementation for the remote X11 server nxagent (used as x2goagent in X2Go). However, this is unfortunately still work in progress, because various theoretical monitor layout issues became evident that require being handled in the new code before it can get merged into nx-libs's current 3.6.x branch. Also, I managed to do some little work on https://arctica-project.org, the still too rudimentary project homepage. read more

30 September 2015

Chris Lamb: Free software activities in September 2015

Inspired by Rapha l Hertzog, here is a monthly update covering a large part of what I have been doing in the free software world:
Debian The Reproducible Builds project was also covered in depth on LWN as well as in Lunar's weekly reports (#18, #19, #20, #21, #22).
Uploads
  • redis A new upstream release, as well as overhauling the systemd configuration, maintaining feature parity with sysvinit and adding various security hardening features.
  • python-redis Attempting to get its Debian Continuous Integration tests to pass successfully.
  • libfiu Ensuring we do not FTBFS under exotic locales.
  • gunicorn Dropping a dependency on python-tox now that tests are disabled.



RC bugs


I also filed FTBFS bugs against actdiag, actdiag, bangarang, bmon, bppphyview, cervisia, choqok, cinnamon-control-center, clasp, composer, cpl-plugin-naco, dirspec, django-countries, dmapi, dolphin-plugins, dulwich, elki, eqonomize, eztrace, fontmatrix, freedink, galera-3, golang-git2go, golang-github-golang-leveldb, gopher, gst-plugins-bad0.10, jbofihe, k3b, kalgebra, kbibtex, kde-baseapps, kde-dev-utils, kdesdk-kioslaves, kdesvn, kdevelop-php-docs, kdewebdev, kftpgrabber, kile, kmess, kmix, kmldonkey, knights, konsole4, kpartsplugin, kplayer, kraft, krecipes, krusader, ktp-auth-handler, ktp-common-internals, ktp-text-ui, libdevice-cdio-perl, libdr-tarantool-perl, libevent-rpc-perl, libmime-util-java, libmoosex-app-cmd-perl, libmoosex-app-cmd-perl, librdkafka, libxml-easyobj-perl, maven-dependency-plugin, mmtk, murano-dashboard, node-expat, node-iconv, node-raw-body, node-srs, node-websocket, ocaml-estring, ocaml-estring, oce, odb, oslo-config, oslo.messaging, ovirt-guest-agent, packagesearch, php-svn, php5-midgard2, phpunit-story, pike8.0, plasma-widget-adjustableclock, plowshare4, procps, pygpgme, pylibmc, pyroma, python-admesh, python-bleach, python-dmidecode, python-libdiscid, python-mne, python-mne, python-nmap, python-nmap, python-oslo.middleware, python-riemann-client, python-traceback2, qdjango, qsapecng, ruby-em-synchrony, ruby-ffi-rzmq, ruby-nokogiri, ruby-opengraph-parser, ruby-thread-safe, shortuuid, skrooge, smb4k, snp-sites, soprano, stopmotion, subtitlecomposer, svgpart, thin-provisioning-tools, umbrello, validator.js, vdr-plugin-prefermenu, vdr-plugin-vnsiserver, vdr-plugin-weather, webkitkde, xbmc-pvr-addons, xfsdump & zanshin.

5 August 2015

Don Armstrong: Introducing dqsub

I've been using qsub for a while now on the cluster here at the IGB at UofI. qsub is a command line program which is used to submit jobs to a scheduler to eventually be run on one (or more) nodes of a cluster. Unfortunately, qsub's interface is horrible. It requires that you write a shell script for every single little thing you run, and doesn't do simple things like providing defaults or running multiple jobs at once with slightly different arguments. I've dealt with this for a while using some rudimentary shell scripting, but I finally had enough. So instead, I wrote a wrapper around qsub called dqsub. What used to require a complicated invocation like:
echo -e '#!/bin/bash\nmake foo'  \
 qsub -q default -S /bin/bash -d $(pwd) \
  -l mem=8G,nodes=1:ppn=4 -;
can now be run with
dqsub --mem 8G --ppn 4 make foo;
Want to run some command in every single directory which starts with SRX? That's easy:
ls -1 SRX* dqsub --mem 8G --ppn 4 --array chdir make bar;
Want instead to behave like xargs but do the same thing?
ls -1 SRX* dqsub --mem 8G --ppn 4 --array xargs make bar -C;
Now, this wrapper isn't complete yet, but it's already more than enough to do what I require, and has saved me quite a bit of time already. You can steal dqsub for yourself Feel free to request specific features, too.

12 October 2014

Iustin Pop: Day trip on the Olympic Peninsula

Day trip on the Olympic Peninsula TL;DR: drove many kilometres on very nice roads, took lots of pictures, saw sunshine and fog and clouds, an angry ocean and a calm one, a quiet lake and lots and lots of trees: a very well spent day. Pictures at http://photos.k1024.org/Daytrips/Olympic-Peninsula-2014/. Sometimes I travel to the US on business, and as such I've been a few times in the Seattle area. Until this summer, when I had my last trip there, I was content to spend any extra days (weekend or such) just visiting Seattle itself, or shopping (I can spend hours in the REI store!), or working on my laptop in the hotel. This summer though, I thought - I should do something a bit different. Not too much, but still - no sense in wasting both days of the weekend. So I thought maybe driving to Mount Rainier, or something like that. On the Wednesday of my first week in Kirkland, as I was preparing my drive to the mountain, I made the mistake of scrolling the map westwards, and I saw for the first time the Olympic Peninsula; furthermore, I was zoomed in enough that I saw there was a small road right up to the north-west corner. Intrigued, I zoomed further and learned about Cape Flattery ( the northwestern-most point of the contiguous United States! ), so after spending a bit time reading about it, I was determined to go there. Easier said than done - from Kirkland, it's a 4h 40m drive (according to Google Maps), so it would be a full day on the road. I was thinking of maybe spending the night somewhere on the peninsula then, in order to actually explore the area a bit, but from Wednesday to Saturday it was a too short notice - all hotels that seemed OK-ish were fully booked. I spent some time trying to find something, even not directly on my way, but I failed to find any room. What I did manage to do though, is to learn a bit about the area, and to realise that there's a nice loop around the whole peninsula - the 104 from Kirkland up to where it meets the 101N on the eastern side, then take the 101 all the way to Port Angeles, Lake Crescent, near Lake Pleasant, then south toward Forks, crossing the Hoh river, down to Ruby Beach, down along the coast, crossing the Queets River, east toward Lake Quinault, south toward Aberdeen, then east towards Olympia and back out of the wilderness, into the highway network and back to Kirkland. This looked like an awesome road trip, but it is as long as it sounds - around 8 hours (continuous) drive, though skipping Cape Flattery. Well, I said to myself, something to keep in mind for a future trip to this area, with a night in between. I was still planning to go just to Cape Flattery and back, without realising at that point that this trip was actually longer (as you drive on smaller, lower-speed roads). Preparing my route, I read about the queues at the Edmonds-Kingston ferry, so I was planning to wake up early on the weekend, go to Cape Flattery, and go right back (maybe stop by Lake Crescent). Saturday comes, I - of course - sleep longer than my trip schedule said, and start the day in a somewhat cloudy weather, driving north from my hotel on Simonds Road, which was quite nicer than the usual East-West or North-South roads in this area. The weather was becoming nicer, however as I was nearing the ferry terminal and the traffic was getting denser, I started suspecting that I'll spend a quite a bit of time waiting to board the ferry. And unfortunately so it was (photo altered to hide some personal information): Waiting for the ferry. The weather at least was nice, so I tried to enjoy it and simply observe the crowd - people were looking forward to a weekend relaxing, so nobody seemed annoyed by the wait. After almost half an hour, time to get on the ferry - my first time on a ferry in US, yay! But it was quite the same as in Europe, just that the ship was much larger. Once I secured the car, I went up deck, and was very surprised to be treated with some excellent views: Harbour view Looking towards the sun   and away from it The crossing was not very short, but it seemed so, because of the view, the sun, the water and the wind. Soon we were nearing the other shore; also, see how well panorama software deals with waves :P! Near the other shore And I was finally on the "real" part of the trip. The road was quite interesting. Taking the 104 North, crossing the "Hood Canal Floating Bridge" (my, what a boring name), then finally joining the 101 North. The environment was quite varied, from bare plains and hills, to wooded areas, to quite dense forests, then into inhabited areas - quite a long stretch of human presence, from the Sequim Bay to Port Angeles. Port Angeles surprised me: it had nice views of the ocean, and an interesting port (a few big ships), but it was much smaller than I expected. The 101 crosses it, and in less than 10 minutes or so it was already over. I expected something nicer, based on the name, but Anyway, onwards! Soon I was at a crossroads and had to decide: I could either follow the 101, crossing the Elwha River and then to Lake Crescent, then go north on the 113/112, or go right off 101 onto 112, and follow it until close to my goal. I took the 112, because on the map it looked "nicer", and closer to the shore. Well, the road itself was nice, but quite narrow and twisty here and there, and there was some annoying traffic, so I didn't enjoy this segment very much. At least it had the very interesting property (to me) that whenever I got closer to the ocean, the sun suddenly disappeared, and I was finding myself in the fog: Foggy road So my plan to drive nicely along the coast failed. At one point, there was even heavy smoke (not fog!), and I wondered for a moment how safe was to drive out there in the wilderness (there were other cars though, so I was not alone). Only quite a bit later, close to Neah Bay, did I finally see the ocean: I saw a small parking spot, stopped, and crossing a small line of trees I found myself in a small cove? bay? In any case, I had the impression I stepped out of the daily life in the city and out into the far far wilderness: Dead trees on the beach Trees growing on a rock Small panorama of the cove There was a couple, sitting on chairs, just enjoying the view. I felt very much intruding, behaving like I did as a tourist: running in, taking pictures, etc., so I tried at least to be quiet . I then quickly moved on, since I still had some road ahead of me. Soon I entered Neah Bay, and was surprised to see once more blue, and even more blue. I'm a sucker for blue, whether sky blue or sea blue , so I took a few more pictures (watch out for the evil fog in the second one): View towards Neah Bay port Sea view from Neah Bay Well, the town had some event, and there were lots of people, so I just drove on, now on the last stretch towards the cape. The road here was also very interesting, yet another environment - I was driving on Cape Flattery Road, which cuts across the tip of the peninsula (quite narrow here) along the Waatch River and through its flooding plains (at least this is how it looked to me). Then it finally starts going up through the dense forest, until it reaches the parking lot, and from there, one goes on foot towards the cape. It's a very easy and nice walk (not a hike), and the sun was shining very nicely through the trees: Sunny forest Sun shinning down Wooden path But as I reached the peak of the walk, and started descending towards the coast, I was surprised, yet again, by fog: Ugly fog again! I realised that probably this means the cape is fully in fog, so I won't have any chance to enjoy the view. Boy, was I wrong! There are three viewpoints on the cape, and at each one I was just "wow" and "aah" at the view. Even thought it was not a sunny summer view, and there was no blue in sight, the combination between the fog (which was hiding the horizon and even the closer islands), the angry ocean which was throwing wave after wave at the shore, making a loud noise, and the fact that even this seemingly inhospitable area was just teeming with life, was both unexpected and awesome. I took here waay to many pictures, here are just a couple inlined: First view at the cape Birds 'enjoying' the weather Foggy shore I spent around half an hour here, just enjoying the rawness of nature. It was so amazing to see life encroaching on each bit of land, even though it was not what I would consider a nice place. Ah, how we see everything through our own eyes! The walk back was through fog again, and at one point it switched over back to sunny. Driving back on the same road was quite different, knowing what lies at its end. On this side, the road had some parking spots, so I managed to stop and take a picture - even though this area was much less wild, it still has that outdoors flavour, at least for me: Waatch River Back in Neah Bay, I stopped to eat. I had a place in mind from TripAdvisor, and indeed - I was able to get a custom order pizza at "Linda's Woodfired Kitchen". Quite good, and I ate without hurry, looking at the people walking outside, as they were coming back from the fair or event that was taking place. While eating, a somewhat disturbing thought was going through my mind. It was still early, around two to half past two, so if I went straight back to Kirkland I would be early at the hotel. But it was also early enough that I could - in theory at least - still do the "big round-trip". I was still rummaging the thought as I left On the drive back I passed once more near Sekiu, Washington, which is a very small place but the map tells me it even has an airport! Fun, and the view was quite nice (a bit of blue before the sea is swallowed by the fog): Sekiu view After passing Sekiu and Clallam Bay, the 112 curves inland and goes on a bit until you are at the crossroads: to the left the 112 continues, back the same way I came; to the right, it's the 113, going south until it meets the 101. I looked left - remembering the not-so-nice road back, I looked south - where a very appealing, early afternoon sun was beckoning - so I said, let's take the long way home! It's just a short stretch on the 113, and then you're on the 101. The 101 is a very nice road, wide enough, and it goes through very very nice areas. Here, west to south-west of the Olympic Mountains, it's a very different atmosphere from the 112/101 that I drove on in the morning; much warmer colours, a bit different tree types (I think), and more flat. I soon passed through Forks, which is one of the places I looked at when searching for hotels. I did so without any knowledge of the town itself (its wikipedia page is quite drab), so imagine my surprise when a month later I learned from a colleague that this is actually a very important place for vampire-book fans. Oh my, and I didn't even stop! This town also had some event, so I just drove on, enjoying the (mostly empty) road. My next planned waypoint was Ruby Beach, and I was looking forward to relaxing a bit under the warm sun - the drive was excellent, weather perfect, so I was watching the distance countdown on my Garmin. At two miles out, the "Near waypoint Ruby Beach" message appeared, and two seconds later the sun went out. What the I was hoping this is something temporary, but as I slowly drove the remaining mile I couldn't believe my eyes that I was, yet again, finding myself in the fog I park the car, thinking that asking for a refund would at least allow me to feel better - but it was I who planned the trip! So I resigned myself, thinking that possibly this beach is another special location that is always in the fog. However, getting near the beach it was clear that it was not so - some people were still in their bathing suits, just getting dressed, so it seems I was just unlucky with regards to timing. However, I the beach itself was nice, even in the fog (I later saw online sunny pictures, and it is quite beautiful), the the lush trees reach almost to the shore, and the way the rocks are sitting on the beach: A lonely dinghy Driftwood  and human construction People on the beach Since the weather was not that nice, I took a few more pictures, then headed back and started driving again. I was soo happy that the weather didn't clear at the 2 mile mark (it was not just Ruby Beach!), but alas - it cleared as soon as the 101 turns left and leaves the shore, as it crosses the Queets river. Driving towards my next planned stop was again a nice drive in the afternoon sun, so I think it simply was not a sunny day on the Pacific shore. Maybe seas and oceans have something to do with fog and clouds ! In Switzerland, I'm very happy when I see fog, since it's a somewhat rare event (and seeing mountains disappearing in the fog is nice, since it gives the impression of a wider space). After this day, I was a bit fed up with fog for a while Along the 101 one reaches Lake Quinault, which seemed pretty nice on the map, and driving a bit along the lake - a local symbol, the "World's largest spruce tree". I don't know what a spruce tree is, but I like trees, so I was planning to go there, weather allowing. And the weather did cooperate, except that the tree was not so imposing as I thought! In any case, I was glad to stretch my legs a bit: Path to largest spruce tree Largest spruce tree, far view Largest spruce tree, closer view Very short path back to the road However, the most interesting thing here in Quinault was not this tree, but rather - the quiet little town and the view on the lake, in the late afternoon sun: Quinault Quinault Lake view The entire town was very very quiet, and the sun shining down on the lake gave an even stronger sense of tranquillity. No wind, not many noises that tell of human presence, just a few, and an overall sense of peace. It was quite the opposite of the Cape Flattery and a very nice way to end the trip. Well, almost end - I still had a bit of driving ahead. Starting from Quinault, driving back and entering the 101, driving down to Aberdeen: Afternoon ride then turning east towards Olympia, and back onto the highways. As to Aberdeen and Olympia, I just drove through, so I couldn't make any impression of them. The old harbour and the rusted things in Aberdeen were a bit interesting, but the day was late so I didn't stop. And since the day shouldn't end without any surprises, during the last profile change between walking and driving in Quinault, my GPS decided to reset its active maps list and I ended up with all maps activated. This usually is not a problem, at least if you follow a pre-calculated route, but I did trigger recalculation as I restarted my driving, so the Montana was trying to decide on which map to route me - between the Garmin North America map and the Open StreeMap one, the result was that it never understood which road I was on. It always said "Drive to I5", even though I was on I5. Anyway, thanks to road signs, and no thanks to "just this evening ramp closures", I was able to arrive safely at my hotel. Overall, a very successful, if long trip: around 725 kilometres, 10h:30m moving, 13h:30m total: Track picture There were many individual good parts, but the overall think about this road trip was that I was able to experience lots of different environments of the peninsula on the same day, and that overall it's a very very nice area. The downside was that I was in a rush, without being able to actually stop and enjoy the locations I visited. And there's still so much to see! A two nights trip sound just about right, with some long hikes in the rain forest, and afternoons spent on a lake somewhere. Another not so optimal part was that I only had my "travel" camera (a Nikon 1 series camera, with a small sensor), which was a bit overwhelmed here and there by the situation. It was fortunate that the light was more or less good, but looking back at the pictures, how I wish that I had my "serious" DSLR So, that means I have two reasons to go back! Not too soon though, since Mount Rainier is also a good location to visit . If the pictures didn't bore you yet, the entire gallery is on my smugmug site. In any case, thanks for reading!

12 September 2014

Elena 'valhalla' Grandi: Sometimes apologies are the worst part

I am sick of hearing apologies directed to me just after a dirty joke.

Usually, I don't mind dirty jokes in themselves: I *do* have a dirty mind and make my share of those, or laugh for the ones my friends with even dirtier minds make.

There are of course times and places where they aren't appropriate, but I'd rather live in a world where they are the exception (although a common one) rather than the norm.

Then there is the kind of dirty jokes strongly based on the assumptions that women are sexual objects, a nuisance that must be tolerated because of sex or both: of course I don't really find them fun, but that's probably because I'm a nuisance and I don't even give them reason to tolerate me. :)

Even worse that those, however, is being singled out as the only women in the group with an empty apology for the joke (often of the latter kind): I know you are not really sorry, you probably only want to show that your parents taught you not to speak that way in front of women, but since I'm intruding in a male-only group I have to accept that you are going to talk as appropriate for it.

P.S. no, the episode that triggered this post didn't happen in a Debian environment, it happened in Italy, in a tech-related environment (and not in a wanking club for penis owners, where they would have every right to consider me an intruder).

25 July 2014

Gunnar Wolf: Nice read: The Fasinatng Frustrating Fascinating History of Autocorrect

A long time ago, I did some (quite minor!) work on natural language parsing. Most of what I got was the very basic rudiments on what needs to be done to begin with. But I like reading some texts on the subject every now and then. I am also a member of the ACM Association for Computing Machinery. Most of you will be familiar with it, it's one of the main scholarly associations for the field of computing. One of the basic perks of being an ACM member is the subscription to a very nice magazine, Communications of the ACM. And, of course, although I enjoy the physical magazine, I like reading some columns and articles as they appear along the month using the RSS feeds. They also often contain pointers to interesting reads on other media As happened today. I found quite a nice article, I think, worth sharing with whoever thinks I have interesting things to say. They published a very short blurb titled The Fasinatng Frustrating Fascinating History of Autocorrect. I was somewhat skeptical reading it links to an identically named article, published in Wired. But gave it a shot, anyway... The article follows a style that's often abused and not very amusing, but I think was quite well done: The commented interview. Rather than just drily following through an interview, the writer tells us a story about that interview. And this is the story of Gideon Lewis-Kraus interviewing Dean Hachamovitch, the creator of the much hated (but very much needed) autocorrect feature that appeared originally in Microsoft Word. The story of Hachamovitch's work (and its derivations, to the much maligned phone input predictors) over the last twenty-something years is very light to read, very easy to enjoy. I hope you find it as interesting as I did.

22 April 2014

Steve Kemp: I've not commented on security for a while

Unless you've been living under a rock, or in a tent (which would make me slightly jealous) you'll have heard about the recent heartbleed attack many times by now. The upshot of that attack is that lots of noise was made about hardening things, and there is now a new fork of openssl being developed. Many people have commented about "hardening Debian" in particular, as well as random musing on hardening software. One or two brave souls have even made noises about auditing code. Once upon a time I tried to setup a project to audit Debian software. You can still see the Debian Security Audit Project webpages if you look hard enough for them. What did I learn? There are tons of easy security bugs, but finding the hard ones is hard. (If you get bored some time just pick your favourite Editor, which will be emacs, and look how /tmp is abused during the build-process or in random libraries such as tramp [ tramp-uudecode].) These days I still poke at source code, and I still report bugs, but my enthusiasm has waned considerably. I tend to only commit to auditing a package if it is a new one I install in production, which limits my efforts considerably, but makes me feel like I'm not taking steps into the dark. It looks like I reported only three security isseus this year, and before that you have to go down to 2011 to find something I bothered to document. What would I do if I had copious free time? I wouldn't audit code. Instead I'd write test-cases for code. Many many large projects have rudimentary test-cases at best, and zero coverage at worse. I appreciate writing test-cases is hard, because lots of times it is hard to test things "for real". For example I once wrote a filesystem, using FUSE, there are some built-in unit-tests (I was pretty pleased with that, you could lauch the filesystem with a --test argument and it would invoke the unit-tests on itself. No separate steps, or source code required. If it was installed you could use it and you could test it in-situ). Beyond that I also put together a simple filesystem-stress script, which read/wrote/found random files, computes MD5 hashes of contents, etc. I've since seen similar random-filesystem-stresstest projects, and if they existed then I'd have used them. Testing filesystems is hard. I've written kernel modules that have only a single implicit test case: It compiles. (OK that's harsh, I'd usually ensure the kernel didn't die when they were inserted, and that a new node in /dev appeared ;) I've written a mail client, and beyond some trivial test-cases to prove my MIME-handling wasn't horrifically bad there are zero tests. How do you simulate all the mail that people will get, and the funky things they'll do with it? But that said I'd suggest if you're keen, if you're eager, if you want internet-points, writing test-cases/test-harnesses would be more useful than randomly auditing source code. Still what would I know, I don't even have a beard..

13 October 2013

John Goerzen: Why are we still backing up to hardlink farms?

A person can find all sorts of implementations of backups using hardlink trees to save space for incrementals. Some of them are fairly rudimentary, using rsync --link-dest. Others, like BackupPC, are more sophisticated, doing file-level dedup to a storage pool indexed by a hash. While these are fairly space-efficient, they are really inefficient in other ways, because they create tons of directory entries. It would not be surprising to find millions of directory entries consumed very quickly. And while any given backup set can be deleted without impact on the others, the act of doing so can be very time-intensive, since often a full directory tree is populated with every day s backup. Much better is possible on modern filesystems. ZFS has been around for quite awhile now, and is stable on Solaris, FreeBSD and derivatives, and Linux. btrfs is also being used for real workloads and is considered stable on Linux. Both have cheap copy-on-write snapshot operations that would work well with a simple rsync --inplace to achieve the same effect ad hardlink farms, but without all the performance penalties. When creating and destroying snapshots is a virtually instantaneous operation, and the snapshots work at a file block level instead of an entire file level, and preserve changing permissions and such as well (which rsync --link-dest can have issues with), why are we not using it more? BackupPC has a very nice scheduler, a helpful web interface, and a backend that doesn t have a mode to take advantage of these more modern filesystems. The only tool I see like this is dirvish, which someone made patches for btrfs snapshots three years ago that never, as far as I can tell, got integrated. A lot of folks are rolling a homegrown solution involving rsync and snapshots. Some are using zfs send / btrfs send, but those mechanisms require the same kind of FS on the machine being backed up as on the destination, and do not permit excluding files from the backup set. Is this an area that needs work, or am I overlooking something? Incidentally, hats off to liw s obnam. It doesn t exactly do this, but sort of implements its own filesystem with CoW semantics.

5 September 2013

Vincent Sanders: Strive for continuous improvement, instead of perfection.


Kim Collins was perhaps thinking more about physical improvement but his advice holds well for software.

A lot has been written about the problems around software engineers wanting to rewrite a codebase because of "legacy" issues. Experience has taught me that refactoring is a generally better solution than rewriting because you always have something that works and can be released if necessary.

Although that observation applies to the whole of a project, sometimes the technical debt in component modules means they need reimplementing. Within NetSurf we have historically had problems when such a change was done because of the large number of supported platforms and configurations.
HistoryA year ago I implemented a Continuous Integration (CI) solution for NetSurf which, combined with our switch to GIT for revision control, has transformed our process. Making several important refactor and rewrites possible while being confident about the overall project stability.

I know it has been a year because the VPS hosting bill from Mythic turned up and we are still a very happy customer. We have taken the opportunity to extend the infrastructure to add additional build systems which is still within the NetSurf projects means.

Over the last twelve months the CI system has attempted over 100,000 builds including the projects libraries and browser. Each commit causes an attempt to build for eight platforms, in multiple configurations with multiple compilers. Because of this the response time to a commit is dependant on the slowest build slave (the mac mini OS X leopard system).

Currently this means a browser build, not including the support libraries, completes in around 450 seconds. The eleven support libraries range from 30 to 330 seconds each. This gives a reasonable response time for most operations. The worst case involves changing the core buildsystem which causes everything to be rebuilt from scratch taking some 40 minutes.

The CI system has gained capability since it was first set up, there are now jobs that:
DownsidesIt has not all been positive though, the administration required to keep the builds running has been more than expected and it has highlighted just how costly supporting all our platforms is. When I say costly I do not just refer to the physical requirements of providing build slaves but more importantly the time required.

Some examples include:
  • Procuring the hardware, installing the operating system and configuring the build environment for the OS X slaves
  • Getting the toolchain built and installed for cross compilation
  • Dealing with software upgrades and updates on the systems
  • Solving version issues with interacting parts, especially limiting is the lack of JAVA 1.6 on PPC OS X preventing jenkins updates
This administration is not interesting to me and consumes time which could otherwise be spent improving the browser. Though the benefits of having the system are considered by the development team to outweigh the disadvantages.

The highlighting of the costs of supporting so many platforms has lead us to reevaluate their future viability. Certainly the PPC mac os X port is in gravest danger of being imminently dropped and was only saved when the build slaves drive failed because there were actual users.

There is also the question of the BeOS platform which we are currently unable to even build with the CI system at all as it cannot be targeted for cross compilation and cannot run a sufficiently complete JAVA implementation to run a jenkins slave.

An unexpected side effect of publishing every CI build has been that many non developer user are directly downloading and using these builds. In some cases we get messages to the mailing list about a specific build while the rest of the job is still ongoing.

Despite the prominent warning on the download area and clear explanation on the mailing lists we still get complaints and opinions about what we should be "allowing" in terms of stability and features with these builds. For anyone else considering allowing general access to CI builds I would recommend a very clear statement of intent and to have a policy prepared for when when users ignore the statement.
Tools
Using jenkins has also been a learning experience. It is generally great but there are some issues I have which, while not insurmountable, are troubling:
Configuration and history cannot easily be stored in a revision control system.
This means our system has to be restored from a backup in case of failure and I cannot simply redeploy it from scratch.

Job filtering, especially for matrix jobs with many combinations, is unnecessarily complicated.
This requires the use of a single text line "combination filter" which is a java expression limiting which combinations are built. An interface allowing the user to graphically select from a grid similar to the output tables showing success would be preferable. Such a tool could even generate the textural combination filter if thats easier.

This is especially problematic of the main browser job which has options for label (platform that can compile the build), javascript enablement, compiler and frontend (the windowing toolkit if you prefer e.g. linux label can build both gtk and framebuffer). The filter for this job is several kilobytes of text which due to the first issue has to be cut and pasted by hand.

Handling of simple makefile based projects is rudimentary.
This has been worked around mainly by creating shell scripts to perform the builds. These scripts are checked into the repositories so they are readily modified. Initially we had the text in each job but that quickly became unmanageable.

Output parsing is limited.
Fortunately several plugins are available which mitigate this issue but I cannot help feeling that they ought to be integrated by default.

Website output is not readily modifiable.
Instead of perhaps providing a default css file and all generated content using that styling someone with knowledge of JAVA must write a plugin to change any of the look and feel of the jenkins tool. I understand this helps all jenkins instances look like the same program but it means integrating jenkins into the rest of our projects web site is not straightforward.
ConclusionIn conclusion I think the CI system is an invaluable tool for almost any non trivial software project but the implementation costs with current tools should not be underestimated.

25 July 2013

John Goerzen: Development Freedom in New Platforms

I started writing code when I was somewhere around 1st grade, hacking some rudimentary BASIC on a TRS-80. Since then, I ve used more programming languages than I can count; everything from C to Prolog, Python to Haskell, and probably a dozen others. I enjoy diversity in programming languages and tools. I ve had an interest in both web and mobile development for some time, but have done little with either. Reflecting on that lately, I realize that both platforms severely restrict my freedom to use the tools I choose. Even on an old x86 with 640K of RAM, I had choices; BASIC, Pascal, and C would work under DOS. (Plus assembler, but I had yet to learn that at the time.) On Android, my choices are Java or Java. (OK, or something that compiles to Java, but those selections are still limited.) Plus it seems like I d have to use Eclipse, which seems to have taken the kitchen sync crown from Emacs long ago, and not in a good way. On iOS, the choices are Objective C. And on the web, it s JavaScript. The problem with all this is that there s little room for language innovation on those platforms. If predictions about the decline of development on the PC pan out, how will the next advance in programming languages gain traction if the hot spots for development are locked down to three main languages? Will we find ourselves only willing to consider languages that can compile to Java/Dalvik bytecode? And won t that put limits on the possible space for innovation to occur within? Yes, I know about things like Mono and the Android Scripting Environment, which offer some potential for hope, but I think it s still safe to say that these platforms are pretty closed to development in unofficial languages. I don t know of major apps in the Android market written in Python, for instance. I know of efforts such as Ubuntu s, but even they simply lock down development to a different toolkit. There is no choice there of GTK vs. Qt, or such things. Sadly, I don t think it has to be this way. My phone has way more power than my computer did just a few years ago, and that computer was capable of not just executing, but even compiling, code written in all sorts of different languages. But I don t know if there s much we can do to change things. Even with native code on Android, plenty still has to go through Dalvik (at least that s my understanding). If you wanted to write a Python app with a full graphical touch interface on Android, I don t think this would be all that easy of a task.

8 September 2012

Steinar H. Gunderson: Video editing on Linux

Lessons learned from trying to do some simple non-linear video editing on Linux: Kdenlive has reached a point where it's usable. Not good by any means there are glaring problems such as the color management support being very rudimentary, almost no multicore support, general lack of optimization, frequent near-crashes, and so on but usable to the point where you can actually create a video from start to end. I'd say it's very roughly comparable to Premiere 5 was, circa 1998, but it's not something I'd recommend to my mother. slowmoVideo, on the other hand, was not very useful to me for framerate up-conversions; the end results were simply too poor given the general complexity in my scenes. It can seemingly do an okay (but not perfect) job for down-conversion if you're willing to hack it a bit and live with general slowness and demand for disk space. So, well, behold the end result, I guess. If you like frisbee, that is.

17 June 2012

Johannes Schauer: port bootstrap build-ordering tool report 2

A copy of this post is sent to soc-coordination@lists.alioth.debian.org as well as to debian-bootstrap@lists.mister-muffin.de. Diary June 4 I added the first version of basenocycles.ml to git. Given an initial set of cross built packages, it tries to compile as much as possible on the resulting system in multiple rounds. June 5 During June 3, I discovered and error in my program that would only come up when using the Debian Sid package lists as the input:
Fatal error: exception Assert_failure("common/edosSolver.ml", 610, 11)
On this day, June 5, I wrote a minimal test case for this problem. The same day, Pietro figured out that this is a bug in dose which will be fixed in the next release. Begin writing code to figure out how important a binary package is for the further build process. Try to use Depsolver.edos_install to find out what packages are needed to make debhelper available. Restructure basenocycles.ml, exclude source packages that already have been built, still trouble with already existing binary packages and Cudf.mem_installed, comment stuff better. June 6 I wrote some crude code (only estimate, not correct, fixed later) that would give a rough overview of how often a given binary package is directly required as a build dependency. Debhelper came out as the most needed package. It is architecture:all, so it does not have to be built but it has unfulfilled runtime dependencies. To make those packages available, 13 (actually 11, fixed later) packages have to be compiled on Ubuntu Natty. But those packages all (except gettext) require debhelper itself to be built. The first dependency cycle. This dependency cycle (actually, the 12 cycles) can be broken by either cross compiling those source packages or by making them build without debhelper. One goal of the program is to help decide what the easier option is, but this is not yet implemented. To play around a bit, I created the possibility to specify a list of packages that are additionally to the minimal set of cross compiled packages also cross compiled. I added the 13 packages found above to the list, thus making the binary packages they build available. This made debhelper available in the system. As a result, 1625 out of 3339 source packages can be built with just a minimal build system (priority:essential packages plus build-essential) and debhelper available. The next package that blocks the most other source packages from being built is cdbs. The next nine packages in that list also require cdbs so it seems to be the next important package to be available. Pietro's suggestions make me:
 - do not open BootstrapCommon but ExtLib, Common, Algo, Debian
 - do proper option parsing and logging
 - use Debcudf.ignore_essential = true
 - do Debcudf.init_tables (binlist@srclist)
 - use @ with shorter list first
 - use more List.rev_append instead of @
 - use CudfAdd.who_provides to find if a package is available
June 7 Pietro and Patrick suggest that for solving the debhelper cycles, one could provide a minimal debhelper version so that the above list of 12 packages can be built without debhelper. I try to figure out how to get a list of packages that are missing to make a package installable/buildable. This functionality should be provided in dose but I fail to find it. June 8 Lacking a solution of the problem of June 7, I write a mail to Pietro. I start my first graphs in ocaml using the ocamlgraph library. The graph I generate, starts off at a binary package. For each binary package it connects new vertices as its runtime dependencies. If a binary package is not arch:all and also not yet otherwise compiled, its source package is also added. The result is a graph in which set of source packages in it will make the initial package available, if those source packages would be cross compiled. The graph is extended further than the source packages. June 9 I refine a couple of functions, make univ_get_pkg_by_name return the package with the highest version number. I wrote a rather lengthy (1027 words) email to the list that explains my status as of this day. I can create graphviz dot files with ocaml, can create node and edge types and create the graph by an imperative pattern that I saw a lot in Pietro's code. Disjunctions are not yet handled correctly (see mail from June 8). The graphs generated look like the following: http://mister-muffin.de/p/8nyc.png June 11 I write a test case which shows how CudfAdd.who_provides doesnt find virtual packages. Automate the process of finding the packages that, if cross compiled, would make another package available. Add more predicates (identifying source packages) and improve input file reading code. Move build_compile_rounds which compiles as many source packages as it can in multiple rounds on a given minimal system a toplevel function and thereby abstract it more. Create a rudimentary text based menu to choose different actions to take for an analysis. Start writing an extended version of simple_dependency_graph for deeper analysis. Use xdot to show graphs from the text menu. Allow saving those graphs to a file. June 12 Move functionality from the extended version of simple_dependency_graph over to the normal version and delete the extended version. Add the new Closure vertex type. Create extended_dependency_graph which is supposed to not contain single binary package vertices but handle a package and its installation set as one vertex. The output of extended_dependency_graph is optionally reduced to the biggest (non degenerate) strongly connected component. User gets the option of choosing the exploration depth. June 13 Pietro replies to my mail from June 8 but apparently I failed to express myself well enough in my last mail, so I rephrase my question. Pietro replies to my email from June 11 and explains how the effect I see is due to "a nuisance of the debian to cudf encoding". As a result I change my code accordingly. June 14 Another lengthy (1130 words) email to the list. I explain what was done in the past days, what parts work and how they work. I list some rationales on why I did things the way I did them. The most important observation is, that after improving my code again and again, I ended up representing the dependency cycle problem in the same (very similar) way that Pietro suggested in the beginning. This is probably a good discovery. Lots of data of that email is now of only little use as of June 16, I make lots of improvements in correctness. As I dont have an answer to my other email to Pietro from June 13, I implement a very crude way to get an answer to the question of what packages are missing for a package to be available/compileable. I called it flatten_vpkgformula_best_effort and it suffers from many faults including disjunctions and package conflicts. Patrick spots a problem. As a result, I make sure that at no point, the source package of an arch:all package can be listed. June 15 As a reply to my mail from June 13, Pietro creates a new branch in the git and adds the code I needed to get a proper installation set. June 16 As a result of Pietro's efforts from June 15, I make great advancements on all fronts. Details of the current status follow in the next section. Results A big leap was made on June 16 due to Pietro's great help on making me understand how Depsolver.listcheck can be used for my purposes. My difficulties in finding the solution myself are rooted in many parts of the dose framework being poorly commented but Pietro did already a couple of documentation commits whenever things were unclear for me. Using Depsolver.listcheck makes it possible to be more distribution agnostic and I dont have to handle vpkgs, virtual packages and constraints myself anymore. The code also doesnt suffer anymore by wrongly analyzed dependencies and conflicts. The only thing that is not yet taken care of, is that Depsolver.listcheck only chooses one out of several possible installation set. A final version should be able to take into account that a different installation set could provide a better solution. Overall, in comparison to two weeks ago, I can now properly build, traverse and analyze graphs, can choose an installation set properly, understand more about dependencies, closures, dose and ocaml in general. Finding the importance of binary packages for building When calculating how many source packages are depending on the availability of a binary package I originally flattened the pkg.Cudf.depends list twice for a rough overview. This is of course wrong due to disjunctions and conflicts and also doesnt provide a deep dependency inspection. The new method is to calculate an installation set that is necessary to compile a source package for every source package. The resulting list of binary packages is then used to find out how often a binary package appears in an installation set. I see three drawbacks though: Removing simple graph The simple graph which contained single binary and source packages was removed. I realized it doesnt really serve any purpose to look at it. As a result, Bin vertices and InstallDep edges are also not part of the graph anymore. Since it was responsible for generating the list of source packages that have to be cross built to make a package available, I created a new function get_cross_source_packages which uses an installation to serve the same purpose. Fix extended_dependency_graph extended_dependency_graph now uses installation sets for generating the list of packages that is needed to compile a source package or install a binary package. The list of build dependencies does not include packages that are already installable. The list of runtime dependencies does not include packages that are otherwise available (cross built, arch:all...). Instead of checking for list membership all the time, I created hash tables for the list of installable as well as for the list of available binary packages. Future There are two big tasks for the next two weeks: Task one is to find a way to give hints on which packages to best consider for having reduced build dependencies. This would then probably finally make use of Pietro's cycle algorithms. Task two is to find a way to break cycles and create a build-DAG from a list of packages that already have staged build dependency information. Patrick is currently working on patching dpkg with Build-Depends-StageN dependencies as making perl cross compilable. If he doesnt need the ability to decide which packages to modify to have staged build dependencies in the near future, then task one is probably less urgent and therefor of less importance right now? On the other hand, I can easily generate fake reduced build dependencies so that doing task two right now would not be a problem. Also, having the solution for task two will make it possible to show the user what effect it would have to add reduced build dependencies to a package. For the reasons above (it's not urgent, task one profits from task two being solved) I will go and implement task two first (if there are no objections from my mentors). Another idea, that I discussed with wookey and Patrick yesterday, was that due to multiarch being used for more and more packages, there should exist a set of packages that is cross compilable without any change to the package. We agreed that I make a list of packages that, if cross compiled, would break dependency cycles and make other packages available. I created such a list of about 160 packages for Ubuntu Natty that, if cross compiled, made it possible to have 87% of Natty available (those numbers have to be treated with caution as I did not yet use the proper way of installation sets when generating that list, but the order of magnitude should be correct). Wookey can then try to cross compile those packages. If some packages of those "crucial" source packages are found to be cross compilable, then they should be cross compiled because it means that no work has to be done to break some cycles. Cross compiling all packages that are cross compilable out of the box is no solution, as only natively compiled packages can go into the archive. This is why the list of potentially additionally cross compiled source packages has to be kept as small as possible.

Next.

Previous.